E  CENTERS 


DAHLGREN 


Navigating  Beyond  the  SLOC: 

Exploring  Improvements  in  Software  Cost  Estimating 


Naval  Postgraduate  School 
Acquisition  Symposium 

17  May  2012 

Kathlyn  H.  Loudin 

Naval  Surface  Warfare  Center,  Dahlgren  Division 


Approved  for  public  release;  distribution  is  unlimited. 

The  views  expressed  in  this  brief  are  those  of  the  authors  and  do  not  reflect  the  official  policy  or 
position  of  the  Department  of  the  Navy ;  the  Department  of  Defense,  or  the  U.S.  government. 


NSWCDD/PN-12/202 


Report  Documentation  Page 

Form  Approved 

OMB  No.  0704-0188 

Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 

VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 

1.  REPORT  DATE 

17  MAY  2012  2' REPORT  TYPE 

3.  DATES  COVERED 

00-00-2012  to  00-00-2012 

4.  TITLE  AND  SUBTITLE 

Navigating  Beyond  the  SLOC:  Exploring  Improvements  in  Software  Cost 
Estimating 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

6.  AUTHOR(S) 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Naval  Surface  Warfare  Center, Dahlgren  Division, 6149  Welsh  Road,  Suite 
203  , Dahlgren, YA, 22448 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS (ES) 

10.  SPONSOR/MONITOR’S  ACRONYM(S) 

11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

Presented  at  the  9th  Annual  Acquisition  Research  Symposium,  May  16  - 17 

,  2012,  Monterey,  CA 

14.  ABSTRACT 

15.  SUBJECT  TERMS 

16.  SECURITY  CLASSIFICATION  OF:  17.  LIMITATION  OF 

ARSTRATT 

1 8 .  NUMBER  1 9a.  NAME  OF 

OF  PAGES  RESPONSIBLE  PERSON 

a.  REPORT  b.  ABSTRACT  c.  THIS  PAGE  Same  3S 

unclassified  unclassified  unclassified  Report  (SAR) 

17 

Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


WARFARE  CENTERS 


DAHLGREN 


Navigating  Beyond  the  SLOC: 

Exploring  Improvements  in  Software  Cost  Estimating 


2 


PR#  3458  APPROVED  FOR  PUBLIC  RELEASE;  DISTRIBUTION  IS  UNLIMITED 


NSWCDD/PN-12/202 


WARFARE  CENTERS 


DAHLGREN 


Overview 


*  Department  of  Defense  (DoD)  Software  (SW)  cost  estimating 

-  Estimates  that  fully  encompass  development  scope  are  key 

*  Source  Lines  of  Code  (SLOC)-based  methods  don't  make  sense 

-  Analogous  to  estimating  the  complexity  of  a  house  painting  job 
based  on  gallons  of  paint 

*  Other  methods  &  metrics 

-  What  is  used? 

-  What  is  useful? 

-  What  is  mandated? 
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Overview  (Cont'd) 


*  Divergence  in  policy  versus  practice 

—  Are  current  policies  misguided? 

*  Are  non-SLOC-based  alternatives  feasible? 

*  Documents,  specifications,  and  management  reviews 

—  How  do  we  quantify  non-Design  Code  Test  (DCT)  effort? 

*  Findings  and  recommendations  from  two  recent  Naval  Surface 
Warfare  Center,  Dahlgren  Division  (NSWCDD),  studies 

—  Software  Estimating  and  Modeling  (SEAM) 

—  Software  Estimating  Enhancement  Task  (SWEET) 
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SWEET  Fiscal  Year  2011  Study  Approach 


Researched  SW  Policy 


DoD  5000  series 
DCARC/SRDR 
ASN(RDA)  Guidebook 


Reviewed  SW  Guidance 


•  NAVSEA05C 

•  NAVAIR  4.2 

•  NCCA 

•  Air  Force  STSC 

•  GAO 

•  NASA 


Collected  Data 

(via  Data  Calls,  Interviews,  Questionnaire,  Site  Visits) 

•  NSWCDD  SW  Programs  (40+  efforts) 

-  SLBM,  TTWCS,  Mk  160,  NFCS 

-  LCS,  AEGIS,  SSDS 

-  SPS,  SGS/AC,  Next-Gen  Simulation 

•  External  Organizations 

-  China  Lake,  SPAWAR,  NAVAIR 


Identified  Best  Practices 

•  Estimate  SW  Effort  with  appropriate  metric 

•  Align  methodology  with  program  maturity 

•  Institute  EVM  and  Metrics  tracking 

•  Validate  estimating  tools  and  estimates 


GAP  Analysis 


Example: 

EVM  not  uniformly  instituted 


Recommended  Process 
Improvements 

•  Interface  with  the  SW 
Community  of  Practice 

•  More  rigorous  EVM 


Assessed  Current  Practices 

•  Variations  in: 

-  SW  development  process  and  maturity 

-  Metrics  collection  and  tracking 

-  EVM  implementation  and  utilization 

-  SW  estimating  methods  (LOE,  SLOC,  etc.) 


1 

1 

I 

1 

Future 

Implementation 
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SLOC-Centric  Policy  and  Guidance 


*  Policy  does  not  always  make  sense 

-  What  works  on  paper  does  not  always  translate  to  practical, 
real-life  scenarios 

*  For  Acquisition  Categories  (ACATs)  I  and/or  IA  programs,  the 
DoD  5000  series  mandates  contractor  delivery  of  Software 
Resources  Data  Reports  (SRDRs)  through  the  Defense  Cost  and 
Research  Center  (DCARC) 

*  Cost  and  Software  Data  Reporting  (CSDR)  plan  required  for 
ACAT  l/IA  programs 

*  Non-ACAT  programs,  e.g.,  AEGIS,  are  not  required  to 
submit  SRDRs 
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SLOC-Centric  Policy  and  Guidance  (Cont'd) 


*  Assistant  Secretary  of  the  Navy  (Research,  Development,  and 
Acquisition)  [ASN(RDA)]  Guidebook  for  Acquisition  of  Naval 
Software  Intensive  Systems  -  limited  application  by  program 
offices  (function  of  awareness) 

*  Naval  Sea  Systems  Command  (NAVSEA)  Program  Executive 
Office  Integrated  Warfare  Systems  guidance 

-  Detailed  data  requested  (other  than  SRDR)  from  additional  programs, 
e.g.,  ACAT  II,  III,  but 

*  There  is  still  no  mechanism  for  enforcement 

*  There  is  no  Data  Item  Description  (DID)  to  give  contractors  specific 
guidance  and  definitions 


No  single  policy  or  mechanism  exists  to  collect  "good"  data  from  both  large  and  small  programs. 
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SWEET  Findings:  SW— Cost  Estimating 


NSWCDD 

•  Most  projects  use  engineering 
build-up  and  Level  of  Effort 
(LOE)-based  approaches 

•  Few  projects  currently  use 
SLOC-based  methods  or  models, 
e.g.,  COCOMO  II,  SLIM 

9  Some  projects  use  code-counting 
tools,  e.g.,  USC  Universal 
CodeCount*  (UCC) 

•  There  is  no  common  tool, 
standards,  or  approach  across  all 
NSWCDD  projects 


External  Entities 

•  AIR  4.1,  Process  Improvement 
Group-affiliated  projects  use 
detailed  engineering  build-up 

9  AIR  4.2,  Space  and  Naval  Warfare 
Systems  Command  (SPAWAR)  1.6, 
NAVSEA  05C,  and  the  Naval  Center 
for  Cost  Analysis  use  SLOC-based 
methods  or  models 

9  Handbooks  are  descriptive, 
not  prescriptive,  so 
variations  exist 


*  Developed  by  University  of  Southern  California  Center  for  Systems  and  Software  Engineering 
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SWEET  Findings:  SW— Historical  Data 


NSWCDD 

•  Several  projects  collect  historical 
cost  data 

•  Few  projects  utilize  this  data  to 
improve  cost  estimates 

•  No  projects  leverage  cost 
performance  metrics  from 
other  programs,  i.e.,  there  is 
limited  collaboration 

•  No  projects  employ  a  focused 
approach  to  data  analysis 

-  No  cost  estimating 
relationships  ascertained 

-  No  "bidding  database" 


External  Entities 

*  Data  availability  issues  pose  problems 
across  the  board 

-  Processes  are  not  in  place  for  the 
cost  estimating  community  to 
efficiently  capture  execution  data 
from  program  offices 

-  Program  offices  are  unwilling  to 
share  data  outside  their  program 

-  Lack  of  consistency  in  interpretation 

-  CSDR/SRDR:  Reliability  can  be  suspect 

*  Progress  is  being  made  but  is  hindered 
by  lack  of  funding  for  working-capital 
organizations,  and  lack  of  resources  for 
mission-funded  organizations 
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NSWCDD 

*  Few  projects  adhere  to  reporting 
requirements  imposed  on  the  industry, 
e.g.,  rigorous  Earned  Value  Management 
(EVM)  System  and  proposal  bases  of 
estimate 

*  Most  projects  use  a  work  breakdown 
structure  to  estimate,  track,  and  report 
execution  data 

*  Most  projects  use  Microsoft  Project 
or  Excel-based  in-house  tools 

*  Tomahawk,  Submarine  Launch  Ballistic 
Missile  (SLBM),  and  Littoral  Combat 
Ship  (LCS)  Surface  Warfare  (SUW)  have 
the  most  mature  processes 

-  Tomahawk  uses  a  progress  tracking  tool, 
but  its  emphasis  is  tracking  not  estimating, 
and  reports  SLOC  using  SLIM  to  compare 
estimates  to  actuals,  i.e.,  validation 

-  SLBM  uses  a  technical  effort  tracking  tool 
for  both  tracking  and  estimating 


—Tracking  and  Reporting 


External  Entities 

•  A  variety  of  tools  are  used,  but 
relatively  few  achieve  the 
"ideal  state" 

-  The  ideal  state  being  the  capability 
of  comparing  estimates  to  actuals 
while  capturing  incremental 
changes  throughout  the  SW 
acquisition  life  cycle 


In  the  areas  of  cost  estimation, 
historical  data,  and  tracking 
and  reporting,  NSWCDD  has 
challenges  similar  to  those 
of  other  organizations. 


SWEET  Findings:  SW 
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SLOC  is  Only  One  Artifact 


Systems  Engineering 


SW:  Preliminary  Design 


SW:  Detailed  Design 


SW:  Coding 


SLOC-based  planning  and  tracking 
relies  on  one  single  activity 
and  one  artifact 


SW:  Unit-Testing 


System  Development 
Activities  and  Artifacts 


SW  Requirements 


Testing 


System  Integration 

Operational 

System  Testing 

Testing 

COST,  SCHEDULE,  RISK  MANAGEMENT,  METRICS  /  CONFIGURATION  MANAGEMENT 


Milestone 
Review  Events 

System  Requirement 
Review 

System  Design  Review 
Test  Readiness  Review 
Program  Reviews 


Alternative 
Artifacts 


SE  Products 

System  Development  Plan 
Build/Capability  Plan 
Concept  of  Operations 
Spend  Plans 

Safety  and  Security  Specifications 
Hardware  Drawings  and  Specifications 
Human  System  Integration  Specifications 

^System  Level  Requirements 
r Architecture  Diagrams  and/or 
DoD  Architecture  Framework 
Interface  Specifications 


Schedules 

Risks  and  Mitigation 

Metrics 


SW  Products 

Integration  and 

Computer  Software 

Test  Products 

Configuration  Item 

Test  Plans, 

Requirements,  Design 

Procedures, 

SW  Architecture 

and  Reports 

►Source  Code/Executables 

Spend  Plans 

Test  Procedures/Reports 

Schedules 

Spend  Plans 

Risks  and 

Schedules 

Mitigations 

Risks  and  Mitigations 

Metrics 

Metrics 
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Literature  Search:  Alternatives  to  SLOC 


*  Identified  33  methods  for  defining  SW  scope,  categorized 

—  by  SW  development  phase 

—  by  manual  or  auto  counting 

*  Other  than  function  point  analysis,  no  method  using  early 
artifacts  is  uniformly  adopted  across  the  SW  community 

*  No  "one  best  way"  endorsed  by  the  SW  community 

*  SLOC  and  function  points  are  still  the  prevailing  metrics 

*  Additional  research  is  needed  to  fully  assess  applicability 

—  It  is  premature  to  recommend  an  alternative  approach  or  tool  for 
SW  estimating 
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An  Example  of  SW  Interfaces  - 

MH-60R  Integration:  Vehicular  Track  Report 


1.  Manual 
Vehicular  Track 

- » 

|  2.  ACK 

I - 

13.  TN  Assignment 

14.  ACK 


LSE 


3.  Vehicular 
Track  Data 


12.  TN  Assignment 


SQQ-89 


4.  89-MT-005 
(Transmit  Symbol) 


11.  TN  Assignment 


wcs 


10.  C&D-MT-307 
(LAMPS  CTSL 
Notification) 


STM 


8.  TM_New 
VehicularTrac^ 


C&D 


TS 


9.  TS_T rackKey 
Assignment 


5.  WCS-MT-160 
(LAMPS 
Track  Data) 


LEF 


7.  New  LAMPS 
Track  Notification 


6.  SM  NewLAMPSVehicularTrack 


SH-60B  Integration 
MH-60R  Integration 


1.  A-S140 
(Vehicular  Track 
Report) 

2.  S-A203  (ACkT 

1  9.  S-A242 

(TN  Assignment)  ^ 

10.  A-S103  (ACK) 


3.  Vehicular 
Track  Data 


LSE 


8.  TN  Assignment 


CURRENT 
FUTURE 


7.  TS_TrackKey 
Assignment  i 


NewVehicularTrack 
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Source  Code  Analysis  - 
Logical  Lines  of  Code  (Cont'd) 


Interface  Messages  ->  Logical  SLOC  ->  $$ 


*  Selected  tactical  programs  that 
represent  a  range  of  complexity 
and  coding  languages 

—  AEGIS  Advance  Capability  Build 
(ACB)  08 

Complex,  Multimission  Combat 
Management  System  (Java,  C++) 

-  TTWCS 

Tomahawk  Missile  Weapon 
Control  System  (C/C++, 

User  Interface  Language,  Ada) 

-  LCS  SUW  Mission  Package 

Integrated  Warfare  Mission 
Module  (Java) 


a; 

|>r  *  j 

Qv4  'M 

■‘!V. 

| 

ACB  0! 
2+  MSLC 

3 

DC 

TTWCS 
-2  MSLOC 


LCS  SUW  MP 
-300  KSLOC 


U.S.  Navy-Released  Photos 
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SW  Estimating  Relationships  (Preliminary) 
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Navigational  Path  Forward 


*  Develop  draft  Contract  Data  Requirements  List/DID  to 
implement  guidance 

*  Adopt  the  code  counting  standards  used  by  the  UCC  tool  or  adopt 
the  actual  tool 

*  Implement  changes  to  code  counting  practices  in  future 
DoD  contracts 

*  Increase  government  access  to  development  artifacts,  e.g.,  code 
and  associated  LOE  expenditures 

*  Investigate  relationships  for  estimating  non-DCT  efforts 

*  Develop  methods  for  collecting  and  using  actual  costs  and 
other  metrics,  e.g.,  EVM,  Dashboard,  as  part  of  optimal 
project  management 
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Questions? 
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